Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic Error Collection #603

Merged
merged 36 commits into from
Nov 16, 2024
Merged

Automatic Error Collection #603

merged 36 commits into from
Nov 16, 2024

Conversation

bagusnl
Copy link
Member

@bagusnl bagusnl commented Nov 9, 2024

Main Goal

Add a way for Collapse to automatically report errors to a self-hosted instance of SentrySDK compatible server

PR Status :

  • Overall Status : Done
  • Commits : Done
  • Synced to base (Collapse:main) : Yes
  • Build status : OK
  • Crashing : No
  • Bug found caused by PR : 1
    • Attachment length when uploading from sentry (currently LogFile) is always 0. This may fail certain instance type of Sentry server returning BadRequest > Upstream SentrySDK bug

TODO

  • Add hard crash detection Unnecessary
  • Add ExceptionHandler to EVERY METHOD CATCHERs Done

bagusnl and others added 14 commits November 8, 2024 22:00
The server running is using a local dinky server, pls no abooze.

TODO: Implement Sentry catcher to ALL EXCEPTION HOLY THAT IS A LOT
`USEREMOTELOGGING`, automatically enabled on Preview & Debug branches for obvious reasons.

update logger types
Does not initialize or de-initialize the utility.

Note: An app restart may be required when selecting the option
sends it to Sentry so we don't have to refactor all our exception calls
Move SentrySDK init to its own helper
This added current log for when exception is sent to Dsn
1. Redacted username in the log file when its not debug build
2. Properly dispose SentrySdk when toggled off
3. Pipe SentrySdk log to Logger
4. Use correct flusher for SentryHelper.cs
5. Fixed cleaner loop for ExceptionHandler_ForLoop by @neon-nyan

Co-authored-by: Kemal Setya Adhi <[email protected]>
No need for adding Sentry's ExceptionHandler on catcher that has SendException already
@bagusnl bagusnl requested a review from a team November 9, 2024 18:19
Copy link

github-actions bot commented Nov 9, 2024

Qodana for .NET

17 new problems were found

Inspection name Severity Problems
RoslynAnalyzers Project does not enable unsafe blocks 🔶 Warning 11
Method return value is never used (private accessibility) 🔶 Warning 2
Async method invocation without await expression 🔶 Warning 1
Dereference of a possibly null reference. 🔶 Warning 1
Invalid XML documentation comment 🔶 Warning 1
Possible 'System.NullReferenceException' 🔶 Warning 1

💡 Qodana analysis was run in the pull request mode: only the changed files were checked
☁️ View the detailed Qodana report

Contact Qodana team

Contact us at [email protected]

Cryotechnic and others added 9 commits November 10, 2024 22:02
Note:

- Some submodules cannot import SentryHelper, which is in `Hi3Helper.Core`
- Some calls are asynchronous, mostly for the handled exceptions, but also where it felt best to use them, feel free to change them if needed
- Most are categorized as `UnhandledOther`, but if we feel the need to change some, we can, it simply affects tagging on the frontend dashboard.
@bagusnl
Copy link
Member Author

bagusnl commented Nov 12, 2024

Awaiting resolution for getsentry/sentry-dotnet#3747

1. Only uploads log file when exception is unhandled
2. Add cache dir path using Collapse's LocalLow data path
3. Dispose _loopToken before creating a new one to prevent memory leak
4. Reordered LogWriteLine on initialization
5. Release exception redirect event when feature is disabled
Also add a way to disable log uploading. This is due to a bug(?) where log upload length reported as 0 and causes BadRequest error. See getsentry/sentry-dotnet#3747
@bagusnl bagusnl marked this pull request as ready for review November 14, 2024 15:19
@bagusnl
Copy link
Member Author

bagusnl commented Nov 14, 2024

Ready for review

Notes:

  • Sentry does not support NativeAOT due to build script issue until planned 5.0.0 SDK release
  • Source linking seems to be broken inside GlitchTip (nor I got it to work on Bugsink), so no source code directly in dashboard
  • Log upload broken due to length=0, a flag const is introduced in SentryHelper.cs to enable it back
  • My server internet seems to have a misfit atm, dashboard/log uploading might be slow or broken at times
  • None of SentrySDK is blocking, so it won't cause an exception by itself (its all handled on its own). No need for trycatch

@bagusnl
Copy link
Member Author

bagusnl commented Nov 14, 2024

Alternatively, we could try to apply for Sentry's OSS program https://sentry.io/for/open-source/
This has a few downsides though:

  1. The data is sent directly to them, unmanaged. Though their Privacy Policy is clear, but it's still a good consideration.
  2. Some DNS blocker blocks the endpoint for Sentry's own domain, but at that point the user might want to disable them anyway 🤷

Please anyone (devs or users) to state your opinion in this thread

@Cryotechnic
Copy link
Collaborator

Cryotechnic commented Nov 15, 2024

Alternatively, we could try to apply for Sentry's OSS program https://sentry.io/for/open-source/ This has a few downsides though:

  1. The data is sent directly to them, unmanaged. Though their Privacy Policy is clear, but it's still a good consideration.
  2. Some DNS blocker blocks the endpoint for Sentry's own domain, but at that point the user might want to disable them anyway 🤷

Please anyone (devs or users) to state your opinion in this thread

I think IMO it's worth applying for Sentry's OSS program, simply because we're using their SDK. A solution could be that we have 2 endpoints, 1 primary and 1 fallback, but at that point we'd need to implement more checks to select 1 automatically, and that's 2 different platforms so I don't know how pertinent that is.

Overall, applying is worth it, we lose nothing by trying. For point 2, I don't think it's a big deal, we also have ErrorSender so the user can just report it in our Discord if anything (which will probably still happen because people won't read the patch notes).

Edit: Some notable benefits for using Sentry is that we won't need to rely on our own equipment which brings the added benefit that we'll have 100% uptime for the feature.

1. Use Sentry's own backend
Still at testing phase, if any change happens it will be changed again to our self-hosted instance.
2. Add a way to disable via system environment variable
Setting env var "DISABLE_SENTRY" to true or 1 will disable exception handling
3. Omit IP address from being sent
4. Regionalized SentryHandler.cs
5. Updated PRIVACY.md
Copy link
Member

@neon-nyan neon-nyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are lots of lines that need the type check to avoid OperationCancelledException or TaskCancelledException spamming Sentry. Other than that, looks fine for me.

Comment on lines -54 to +57
throw ex.Flatten().InnerExceptions.First();
var innerExceptionsFirst = ex.Flatten().InnerExceptions.First();
SentryHelper.ExceptionHandler(innerExceptionsFirst, SentryHelper.ExceptionType.UnhandledOther);
throw innerExceptionsFirst;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think throwing AggregateException into Sentry will be a good idea since it can actually be another type exception like OperationCancelledException or TaskCancelledException. In my honest opinion, I think we need to check the innerExceptionsFirst first to check if it's no other than OperationCancelledException or TaskCancelledException, then just throw it. Other than that, just ignore it

Comment on lines -76 to +79
throw ex.Flatten().InnerExceptions.First();
var innerExceptionsFirst = ex.Flatten().InnerExceptions.First();
SentryHelper.ExceptionHandler(innerExceptionsFirst, SentryHelper.ExceptionType.UnhandledOther);
throw innerExceptionsFirst;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Comment on lines -106 to +109
throw ex.Flatten().InnerExceptions.First();
var innerExceptionsFirst = ex.Flatten().InnerExceptions.First();
SentryHelper.ExceptionHandler(innerExceptionsFirst, SentryHelper.ExceptionType.UnhandledOther);
throw innerExceptionsFirst;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Comment on lines -339 to 346
catch
catch (Exception ex)

{
await SentryHelper.ExceptionHandlerAsync(ex, SentryHelper.ExceptionType.UnhandledOther);
IsRunning = false;
throw;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This catch might probably throw OperationCancelledException or TaskCancelledException too due to the usage of Game Repair mechanism for checking the game integrity before the delta-patch come into play.

Comment on lines 428 to 432
catch (Exception ex)
{
await SentryHelper.ExceptionHandlerAsync(ex, SentryHelper.ExceptionType.UnhandledOther);
LogWriteLine($"Error has occurred while performing delta-patch!\r\n{ex}", LogType.Error, true);
throw;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This catch might also throw OperationCancelledException or TaskCancelledException. Type check is needed to exclude these two cancellation exceptions from sentry

Comment on lines +60 to +63
catch (Exception ex)
{
await SentryHelper.ExceptionHandlerAsync(ex, SentryHelper.ExceptionType.UnhandledOther);
throw;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need cancellation exception check

Comment on lines +285 to 287
await SentryHelper.ExceptionHandlerAsync(ex, SentryHelper.ExceptionType.UnhandledOther);
LogWriteLine($"Failed while getting CDN content from: {cdnProp.Name} (prefix: {cdnProp.URLPrefix}) (relPath: {relativeURL})\r\n{ex}", LogType.Error, true);
return null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Comment on lines +314 to 316
await SentryHelper.ExceptionHandlerAsync(ex, SentryHelper.ExceptionType.UnhandledOther);
LogWriteLine($"Failed while getting CDN content from: {cdnProp.Name} (prefix: {cdnProp.URLPrefix}) (relPath: {relativeURL})\r\n{ex}", LogType.Error, true);
return false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Comment on lines +366 to 368
await SentryHelper.ExceptionHandlerAsync(ex, SentryHelper.ExceptionType.UnhandledOther);
LogWriteLine($"Failed while getting CDN content from: {cdnProp.Name} (prefix: {cdnProp.URLPrefix}) (relPath: {relativeURL})\r\n{ex}", LogType.Error, true);
return false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Comment on lines +430 to 432
await SentryHelper.ExceptionHandlerAsync(ex, SentryHelper.ExceptionType.UnhandledOther);
LogWriteLine($"CDN content from: {cdnProp.Name} (prefix: {cdnProp.URLPrefix}) (relPath: {relativeURL}) has failed to initialize due to an exception:\r\n{ex}", LogType.Error, true);
return CDNUtilHTTPStatus.CreateInitializationError();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

1. Use the InnerException if exception caught is an AggregateException
2. Throw TCE/OCE to the bin and not upload them
3. Deduplicate exception handling mechanism and use an inner method instead
@bagusnl
Copy link
Member Author

bagusnl commented Nov 16, 2024

@neon-nyan

There are lots of lines that need the type check to avoid OperationCancelledException or TaskCancelledException spamming Sentry. Other than that, looks fine for me.

Fixed in 653ed45 by returning all ExceptionHandler method if it sees an OCE/TCE being caught and send them straight out to the bin, also the handler will use the first exception instead if it sees AggregateException being caught

Also set the environment as either Debug/non-debug based if there is any debugger attached
Copy link
Member

@neon-nyan neon-nyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bagusnl
Copy link
Member Author

bagusnl commented Nov 16, 2024

Approval received. Merging to main...
Shall any changes happened with Sentry's OSS license registration happened, I will notify in staff channels.

We also need to notify users about the privacy policy changes and let them know the way to disable sentry's mechanism can be done either by

  1. App Settings
  2. EnvVar DISABLE_SENTRY as true

@bagusnl bagusnl merged commit 7ac974b into main Nov 16, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants